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FOREWORD 


As  part  of  Its  program  on  leadership  development,  the  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  provides  an  Integrated 
research  effort  on  the  Officer  Personnel  Management  System  (OIWS)  and 
Officer  Evaluation  System  (OES)  which  includes  Identifying  operationally 
feasible  career  progression  choice  points  for  evaluation  of  differential 
performance  and  potential  of  Individual  officers,  combining  these  eval¬ 
uation  factors  into  "officer  Indices"  to  be  used  In  assigning  Individuals 
to  specific  career  development  programs  In  OIMS,  and  using  the  officer 
Indices  to  provide  "Order  of  Merit"  listings  for  assignment,  military 
school  selection,  and  promotion.  The  present  publication  Is  concerned 
wi^  the  evaluative  components  used  in  the  Order  of  Merit  List  (OML). 

This  Research  Problem  Review  analyzes  the  statistical  properties  and 
operational  nature  of  the  evaluative  predictor  variables  used  in  the  CM, 
to  determine  its  utility  for  personnel  management  decisions.  It  also 
suggests  appropriate  differential  weights  to  be  applied  to  these  measures 
for  specific  career  management  purposes  _ _ - 

K 

The  entire  task  Is  responsive  to  the  special  requirements  of  the 
Deputy  Chief  of  Staff  for  Personnel  as  well  as  the  objectives  of  RDTE 
Project  2Q7627I7A722 ,  "Indices  for  Officer  Career  Development."  FY  1974- 
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BRIEF 


Requirement : 

To  determine  the  statistical  adequacy  and  appropriate  scoring  weight 
of  each  of  the  evaluative  factors  used  for  the  Order  of  Merit  List  (OML). 

Procedure: 

Individual  records  for  838  Infantry  and  Air  Defense  officers  being 
considered  for  nomination  to  Conmand  and  General  Staff  College  (G4GSC) 
and  Senior  Service  College  (SSC)  were  analyzed.  The  statistical  aspects 
investigated  were  the  weights  used  for  scoring,  the  measurement  charac¬ 
teristics,  and  the  selection  factor  relationships. 

Findings : 

These  findings  should  be  considered  tentative  because  of  the  compara¬ 
tively  small  sample  and  inclusion  of  only  two  Army  branches — both  combat 
arms . 

Scores  should  be  standardized  for  all  selection  factors  in  order 
that  weights  used  match  the  weights  intended  by  management. 

Military  Education  might  be  eliminated  as  part  of  the  OML  because 
most  officers  have  equal  education  levels. 

The  present  procedures  for  scoring  the  OER  should  be  changed  to 
eliminate  an  artificial  and  unreliable  spread  in  scores  that  could  lead 
to  an  erroneous  interpretation. 

Use  of  Branch  Subjective  scores  should  be  reviewed;  such  scores  seem 
to  be  too  closely  related  to  each  other  and  to  other  similar  scores. 

Differences  in  distribution  of  evaluative  variables  between  branches 
suggest  that  branch-specific  OML  scoring  might  be  investigated.  Differ¬ 
ences  found  in  the  weights  for  C&GSC  and  SSC  suggest  that  further 
changes  in  selection  factors  for  the  two  programs  should  be  investigated. 


Utilization  of  Findings: 

The  results  of  the  analysis  suggest  that  the  operational  measures  of 
performance  and  potential  used  in  the  OML  can  be  improved.  The  research 
provides  tentative  estimates  of  the  discriminatory  value  of  the  separate 
measures  used  for  various  decisions  and  suggests  procedures  for  devel¬ 
oping  weights  to  apply  to  these  measures  when  combined  for  specific 
career  management  purposes. 
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BACKGROUND  AND  PURPOSE 

Because  the  number  of  students  must  be  limited,  the  selection  of 
Army  officers  for  attendance  at  Command  and  General  Staff  Colleges 
(C&GSC)  and  Senior  Service  Colleges  (SSC)  generates  a  need  for  an  objec¬ 
tive  and  quantifiable  Army  wide  system  to  determine  on  the  basis  of 
merit  those  officers  nominated  for  attendance,  and  rank  ordering  the 
eligible  officers. 

The  Military  Personnel  Center  (MILPERCEN) ,  in  response  to  this 
requirement,  has  developed  a  set  of  procedures-- "Order  of  Merit  Lists" 
(OML)--for  establishing  a  numerical  value  for  the  quality  of  the  record 
of  each  eligible  officer,  and  then  on  the  basis  of  the  numerical  values 
arraying  the  eligible  officers  on  a  rank  order  roster.  While  these  OML 
procedures  permit  the  quantifying  of  the  records  for  a  numerical  compar¬ 
ison,  they  have  not  been  subjected  to  a  systematic  investigation  of  the 
statistical  adequacy  of  the  record  factors  or  the  numerical  weight  to 
be  applied  to  each  factor. 

This  Investigation  was  undertaken  by  the  Army  Research  Institute  for 
the  Behavioral  and  Social  Sciences  to  identify  the  nature  and  statistical 
properties  of  the  common  evaluative  variables,  as  an  initial  background 
step  in  the  development  and  application  of  "Indices  for  Officer  Career 
Development"  for  a  variety  of  career  management  needs. 

The  investigation  focused  on  three  basic  questions: 

1.  Do  the  scores  assigned  to  each  evaluative  factor  by  career 
management  reflect  the  true  mathematical  weights  forced  by  the  statis¬ 
tical  properties  of  the  variable? 

2.  What  differential  measurement  characteristics  of  the  evaluative 
factors  are  utilized? 

5.  What  are  the  relationships  between  the  various  evaluative 
factors--e. g. ,  independence  of  evaluations  vs.  duplication? 


DESIGN  OF  THE  RESEARCH 

The  research  was  exploratory  in  nature,  and  samples  were  used  from 
only  two  Army  branches  (Infantry  and  Air  Defense)  of  officers  being 
considered  for  nomination  to  Comnand  and  General  Staff  College  (C&GSC) 
and  Senior  Service  College  (SSC).  The  number  and  background  of  officers 
from  each  group  was: 


INFANTRY 


AIR  DEFENSE 


GS:GCS--Prlmarily  majors-- 
beginning  I972 

2S6 

211 

SSC— Primarily  LTC-- 
beginning  I972 

188 

175 

The  statistical  aspects  investigated  were  the  weights  for  scoring,  the 
measurement  characteristics,  and  the  selection  factor  relationships. 
Comparison  of  Weights--Individual  records  were  scored  in  two  ways: 

1)  The  *'real  weights"  used  in  the  present  procedures  by  MILPERCEN  were 
based  upon  the  total  number  of  points  assigned  to  a  selection  factor. 

2)  Weights  for  scoring  were  based  upon  first  equalizing  all  the  selection 
factors  (standardizing)  and  then  applying  (multiplying)  the  "desired 
weights"  to  them.  Measurement  Characterlstics--This  included  the  com¬ 
pilation  of  the  average  and  the  distribution  of  scores.  Selection  Factor 
Relationships --For  each  of  the  four  groups,  the  extent  to  which  being 
high  (or  low)  on  one  factor  was  related  to  being  high  (or  low)  on  another 
factor  was  investigated. 


RESEARCH  FINDINGS 

Tables  1  and  2  in  the  appendix  summarize  the  weights  applied  to  each 
factor  of  the  OML  by;  1)  The  present  system  "real  weights"  determined 
by  the  distribution  of  scores  for  a  factor  rather  than  by  management 
intention,  and  2)  a  standardized  score  system  (desired  weights)  which 
reproduced  the  weights  as  intended  by  MILPERCEN. 

The  difference  between  "real"  and  "desired"  weight  for  a  given 
variable  reflects  the  degree  to  which  that  variable  is  over-  or  under¬ 
represented  in  the  composite  score  compared  to  its  intended  represen¬ 
tation  by  career  management. 

Results  for  AD  officers  eligible  for  C&G3C  (Table  1--Appendix)  show 
that  the  OER  67-6  and  "Performance  in  a  difficult  assignment"  both 
receive  a  weight  in  the  final  OML  greater  than  intended.  Conversely, 
Military  Education,  OER  67-5,  '^Manner  of  performance  (MOP)  in 
highest  command"  all  showed  a  decrease  in  weight  ("real")  below  the 
desired  level.  The  results  for  AD  of ficers--SSC  (Table  1)  indicated 
that  OER  67“6,  Command  MOP,  Branch  Subjective,  and  Performance  in  a 
difficult  assignment  all  have  higher  "real"  weight  than  desired  with 
decreases  being  found  for  Military  Education,  OER  and  Staff  MOP. 

Results  for  the  IN  branch  (Table  2--Appendix)  were  similar  to  AD  for 
both  programs  (IN  branch  did  not  include  subjective  evaluations  in  its 
OML  procedures  for  either  sample),  with  OER  C7-6  and  OER  67-5  being 
high  and  Military  Education  and  Staff  MOP  being  low.  For  IN  officers 
(GS;GSC)  the  Command  MOP  was  also  lower. 
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While  large  differences  were  found  between  the  "real"  and  "desired" 
weights,  It  should  be  noted  that  the  OML  order  produced  by  each 
weighting  system  was  very  similar  (correlations  between  each  list  were 
In  the  .90's).  But  this  is  not  an  unusual  finding  for  weighting  problems, 
due  to  mathematical  constraints. 

Table  3  In  the  appendix  lists  the  means  and  standard  deviations  for 
each  OML  factor.  Scores  on  the  CWL  reflect  an  arbitrary  numeric  value 
assigned  to  the  various  categories;  i.e.,  for  Civilian  Education  the 
score  is  the  "number"  assigned  to  the  level  of  education  and  Is  not  the 
number  of  years.  All  OML  factors  were  assigned  values  (and  ranges )  as 
specified  in  the  Operating  Instruction,  614-100  (22),  of  the  Officer 
Personnel  Directorate,  26  May  I972.  The  means  and  standard  deviations 
for  Command  MOP  and  civilian  education  are  acceptable  approximations  of 
a  normal  curve.  While  the  means  and  standard  deviations  for  OER  67-6 
and  OER  67-^  were  acceptable,  this  was  achieved  by  a  highly  questionable 
procedure.  The  original  OER  scores  were  transformed  by  converting  to 
a  percentile  distribution,  where  the  highest  score  receives  a  percentile 
score  of  100,  the  lowest  score  a  zero,  and  the  remainder  spread  evenly 
throughout  the  range.  The  percentile  score  is  then  transformed  (by 
conversion  table)  back  into  an  "OML  OER"  score,  and  a  one  point  differ¬ 
ence  In  the  original  Increased  to  a  three  point  difference  In  the  OML 
score.  Therefore,  if  two  officers  have  a  raw  score  difference  of  two, 
which  Is  not  reliable  (I.e.,  not  truly  different),  the  OML  score  would 
reflect  a  ^  point  difference.  This  five  point  difference  would  be 
viewed,  by  users,  as  a  real  difference  when  In  fact  nothing  has  changed. 
This  Increase  In  differences  led  to  the  large  weights  and  standard 
deviations.  The  standard  deviations  for  Combat  Experience,  Staff  MOP, 
and  Decorations  were  reasonable  but  the  means  tended  to  be  extremely 
high  or  low  thus  producing  a  skewed  distribution  for  these  variables. 

For  the  remainder  of  the  OML  factors--Military  Education,  Branch  Subjec¬ 
tive,  Educational  Achievement,  Special  Advanced  Degree,  and  MOP  In  a 
difficult  assignment — the  means  were  extremely  low  (or  In  the  case  of 
SSC  Military  Education,  high)  with  very  small  standard  deviations.  Also, 
these  same  OML  factors  had  very  low  real  weights.  The  Intercorrelations 
for  AD  are  presented  In  Table  4  and  for  IN  In  Table  ^  (Appendix).  For 
both  groups  of  AD,  Military  Education,  Civilian  Education,  and  Combat 
Experience  do  not  have  significant  correlations  with  the  Total  (final) 

OML  score.  In  addition,  for  the  AD  C&GSC  group,  the  Decoration  score  Is 
not  related  to  Total  OML  score.  A  similar  pattern  emerged  for  both 
groups  of  IN;  Combat  Experience  did  not  have  a  significant  correlation 
with  the  Total  OML  score.  Military  Education  for  the  SSC  group  was  not 
related  to  Total  OML  score,  and  for  the  C&GSC  group  Civilian  Education 
was  not  related  to  total  OML  score. 

The  correlations  between  OER  67-6  and  CCR  67-^  were  very  low,  and 
significant  for  IN  (both  groups)  and  AD  SSC  but  not  significant  for  AD 
C&GSC.  These  relationships  Indicate  that  the  OER  sums  (67-6  and  ^) 
were  not  very  reliable  Indicators  over  time. 
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There  was  a  tendency  for  Branch  Subjective  points  (available  for  AD 
only)  to  be  internally  consistent  (correlated)  with  other  OML  factors 
and  somewhat  consistent  with  each  other.  This  tendency  was  most  pro¬ 
nounced  for  the  SSC  group. 

Finally,  the  two  groups  (SSC  &  C&GSC)  had  different  patterns  of 
correlation  within  a  branch,  and  even  larger  differences  in  correlations 
were  evident  across  branches  and  school  groups.  Recognition  of  these 
similarities  and  differences  might  be  utilized  to  develop  a  more 
differentiated  OML  system. 


CONCLUSIONS  AND  IMPLICATIONS 

The  findings  and  implications  are  summarized  below.  The  data  used 
were  limited,  and  therefore  the  following  conclusions  are  tentative: 

Standard  scores  (same  mean  and  standard  deviation)  for  all  selection 
factors  should  be  used.  Standardization  would  yield  the  weights  desired 
by  career  managers  for  each  factor,  which  current  operational  procedures 
do  not  provide. 

2.  Military  education  should  be  eliminated  as  a  selection  factor  in 
OML's  for  school  selection.  In  most  cases  officers  all  have  the  same 
educational  level;  therefore  inclusion  of  a  numeric  value  does  not  affect 
any  officer's  position  on  the  OML. 

5-  The  present  procedures  for  scoring  the  PER  should  be  changed.  OER 
scoring  includes  the  conversion  to  percentile  scores  which  produces  an 
artificial  spread  in  scores  that  may  well  be  unreliable.  The  increased 
spread  in  the  OML  score  could  lead  to  over- interpretation  of  score 
differences  which  are  not  real  differences. 

4.  The  use  of  Branch  Subjective  evaluations  should  be  reviewed.  Present 
procedures  lead  to  Branch  Subjective  scores  with  reduced  ranges,  highly 
related  to  each  other,  and  related  to  other  similar  »1L  scores. 

5*  The  use  of  branch  specific  OML  scoring  should  be  investigated.  Branch 
analysis  indicated  differences  in  the  distribution  of  evaluative  variables 
which  might  be  used  to  produce  more  meaningful  scores  within  branches. 

6.  Further  changes  in  3>coring  for  C&GSC  and  SSC  should  be  investigated. 
While  current  procedures  differ  somewhat  between  programs  (different 
weights  assigned  to  selection  factors),  results  of  the  analysis  indicated 
that  further  changes  in  selection  factors  may  be  warranted. 


SUMMARY 


The  results  of  this  investigation  have  pointed  to  the  need  for 
changes  in  current  C®iL  procedures  as  outlined  (conclusions  1-5)  above. 
The  limitations  of  sample  sizes  and  of  only  two  branch  groups  (both 
combat  oriented)  will  make  it  necessary  to  confirm  the  indications  and 
trends  found  between  and  within  career  branches  (conclusions  4-6)  on 
larger  samples  from  all  branches,  to  verify  the  conclusions  of  this 
preliminary  analysis. 
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